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SUMMARY 

In  order  for  a  defense  organization  to  gain  understanding  of  their  performance  in  complex  tactical 
situations  it  is  important  to  accurately  model  the  way  opposing  and  own  forces  operate.  It  is  a  significant 
challenge  to  develop  such  models  due  to  the  increasing  interdependence  between  personnel  and 
technology.  To  gain  a  clear  insight  in  how  personnel  and  technology  (or  a  man-machine  network ) 
together  influence  operational  effectiveness  one  has  to  model  environment,  systems  and  human  behavior 
at  a  very  detailed  level.  TNO  is  performing  research  focused  on  maintaining  or  improving  operational 
effectiveness  with  the  introduction  of  new  technologies  such  as  smart  sensors  and  unmanned  systems. 
Simulation  developers  face  the  challenge  of  coupling  environment,  sensor,  system  and  human  behavior 
models  in  order  to  perform  such  research.  There  are  currently  not  many  standards  and  tools  that  support 
this  kind  of  coupling,  especially  in  the  area  of  man-machine  collaboration.  TNO  is  developing  the  Man- 
Man-Machine-Machine  Etiquette  (M4E)  toolbox  that  helps  in  integrating  models  of  man  and  machine  and 
allows  them  to  run  in  concert  in  a  simulated  environment. 


1.0  INTRODUCTION 

Defense  organizations  are  looking  for  ways  to  improve  their  performance.  Much  time  and  effort  is  spent  on 
training  of  personnel,  and  research  and  money  on  the  development  or  acquisition  of  new  technologies.  To 
investigate  in  how  far  these  efforts  lead  to  improved  performance,  experimentation  is  required.  Because  real- 
world  experiments  are  often  difficult  to  organize  due  to  issues  of  safety,  logistics  and  money,  many 
experiments  take  place  in  simulated  environments.  To  gain  a  good  understanding  of  performance  in  complex 
tactical  situations  it  is  important  to  accurately  model  the  way  opposing  and  own  forces  operate.  It  is  a 
significant  challenge  to  develop  such  models  due  to  the  increasing  interdependence  between  personnel  and 
technology.  The  pace  of  technological  innovation  is  still  increasing  exponentially  and  it  is  becoming  more 
difficult  to  leverage  technological  innovation  on  the  battlefield.  Additionally,  due  to  the  increase  of 
technology  in  the  environment  of  the  operator,  the  task  of  operators  is  shifting  from  procedural  oriented  tasks 
towards  resource  management  and  decision  making  tasks.  To  keep  track  of  how  personnel  and  technology 
(or  man-machine  network)  together  influence  operational  effectiveness  one  has  to  model,  environment, 
systems  and  human  behavior  at  a  very  detailed  level.  We  call  the  search  for  the  optimal  way  for  man  and 
machine  to  collaborate  the  search  for  the  optimal  man-man-machine-machine  etiquette  (M4E). 

TNO  is  performing  research  focused  on  improving  or  at  least  maintaining  operational  effectiveness  with 
the  introduction  of  new  technologies  such  as  smart  sensors  and  unmanned,  autonomous  systems.  To 
perform  such  research  in  a  simulated  environment,  simulation  developers  face  the  challenge  of  coupling 
environment,  sensor,  system  and  human  behavior  models.  There  are  currently  not  many  standards  and 
tools  to  support  this  kind  of  coupling,  especially  not  in  the  area  of  man-machine  collaboration.  While  there 
are  standards  for  certain  types  of  models  (e.g.  DIS,  HLA,  FIPA,  TCP/IP,  XML)  1  there  is  no  standard  to 

1  DIS:  IEEE  Standard  1278,  HLA:  IEEE  1516-2010,  FIPA:  www.fipa.org,  TCP/IP:  RFC  1 122,  XML:  www.w3.org/XML/ 
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couple  the  different  types  of  models.  This  means  that  for  each  new  man-machine  network  under 
investigation  the  simulation  developer  has  to  develop  a  simulation  environment  nearly  from  scratch, 
because  existing  systems  are  not  transparent,  not  open  or  simply  too  monolithic.  This  is  hindering  the 
amount  of  re-use  that  is  potentially  possible.  This  in  turn  is  lengthening  development  time  and  therefore 
driving  up  costs  of  research.  Ideally  one  would  just  spend  time  on  measuring  relative  effectiveness  of 
different  uses  of  novel  systems.  Currently  much  effort  goes  into  the  building  of  the  infrastructure  that 
makes  experimenting  possible. 

Because  the  simulation  developers  can  not  easily  re-use  model  couplings,  the  lessons  learned  are  not 
anchored  in  technology.  If  each  new  project  develops  new  simulation  technology  the  wisdom  of  previous 
generations  is  lost.  Therefore  there  is  an  increased  risk  of  repeating  the  same  mistakes.  Additionally,  the 
fact  that  current  interoperability  standards  are  monodisciplinary  (i.e.  coupling  only  simulation,  or  only 
agents)  means  that  there  is  little  opportunity  for  researchers  to  explore  multidisciplinary  approaches. 

TNO  undertakes  multiple  projects  that  require  the  coupling  of  diverse  models.  Therefore,  the  burden  of  the 
previous  mentioned  issues  is  strongly  felt.  To  alleviate  these  issues  TNO  is  developing  the  M4E  toolbox  to 
help  with  integrating  diverse  models  and  to  allow  them  to  run  in  concert  in  a  simulated  environment. 

Each  investigation  into  man-machine  networks  has  a  specific  focus  and  it  is  therefore  not  possible  to 
completely  standardize  the  way  to  build  a  composite  system  of  diverse  models.  We  therefore  propose  not 
to  aim  for  standardization,  but  to  aim  for  facilitation.  Our  goal  is  to  provide  tools  that  offer  a  way  to 
couple  systems  using  standards  if  possible,  to  offer  template  programs  where  it  is  useful,  and  to  leave 
undefined  those  things  that  are  specific  to  a  research  project. 

1.1  Related  Work 

Much  research  has  been  done  on  developing  simulations  that  provide  insight  in  operational  effectiveness 
and  on  how  this  can  be  improved.  Such  research  is  widespread  both  within  TNO  and  internationally.  The 
man-man-machine-machine  etiquette  approach  aims  to  harmonize  a  number  of  research  areas. 

One  of  the  relevant  areas  is  the  relatively  new  research  field  concerned  with  heterogeneous  forms  of 
simulation.  TNO  has  investigated  ways  to  combine  Live  simulations  (real  people  and  real  systems)  with 
Virtual  simulations  (real  people,  simulated  systems)  and  Constructive  simulations  (simulated  people, 
simulated  systems)  [1 1],  Such  a  combination  could  provide  additional  flexibility  of  simulations.  Important 
research  issues  in  this  area  are  how  to  ensure  the  effective  sharing  of  data  between  simulations,  and  how 
simulations  could  best  interoperate. 

A  similar  issue  arises  in  the  area  of  joint  simulation,  in  which  multiple  simulation  platforms,  frequently 
from  various  (parts  of)  defense  organizations,  are  coupled.  Joint  simulation  is  valuable  because  it  can 
create  realistic  training  situations  that  specifically  address  collaboration  issues  between  countries.  Again 
effective  interoperability  of  all  the  participating  simulated  systems  is  key  to  tap  this  potential  value  [12], 

The  same  issue  again  rises  when  combining  separately  developed  simulations  to  prove  a  concept.  At  TNO 
research  is  done  to  test  novel  sensors  (or  sensor  simulations)  and  their  effectiveness  in  operational  settings 
[5].  The  value  of  a  sensor  can  be  determined  by  applying  the  sensor  in  multiple  contexts  (e.g.  does  a 
camera  sensor  work  equally  well  on  land  as  on  water).  Simulating  these  contexts  is  a  cost  effective  way  to 
test  this.  However,  this  requires  that  a  sensor  can  interoperate  with  multiple  simulations. 

In  applied  research  often  such  interoperability  issues  arise,  e.g.  when  asked  to  investigate  a  specific  issue 
as  “how  do  I  best  protect  a  compound”  or  “how  do  I  defend  best  against  mortar  attacks”.  Pragmatics 
demand  that  one  does  not  build  all  required  simulations  from  scratch  but  starts  a  search  for  relevant 
existing  systems.  Next  step  is  to  combine  those  systems  and  start  a  simulation  based  analysis  of  the  issue. 
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When  creating  such  a  composite  simulation  system  again  interoperability  is  a  key  issue.  At  TNO  the 
experience  is  that  even  with  current  available  standards  (i.e.  DIS,  various  HLA  flavors)  it  is  still  a 
significant  effort  to  harmonize  simulations. 

The  above  examples  show  a  trend  towards  increased  use  of  (widely)  distributed  systems.  TNO  is  at  the 
forefront  of  this  development  [13].  For  example,  there  is  an  ongoing  effort  to  investigate  distributed  sensor 
systems  and  sensor  networks.  Such  networks  of  sensors  create  value  because  by  (intelligently)  fusing  the 
sensor  data  more  informative  results  can  be  extracted  out  of  relatively  simple  sensors.  However,  it  is  not 
trivial  to  answer  the  question  on  how  and  what  sensors  should  communicate  amongst  themselves  to  yield 
the  most  useful  and  usable  sensor  data. 

A  specific  example  of  a  distributed  system  is  a  simulation  in  which  agent  technology  is  used  to  simulate 
human  (cognitive)  behavior.  TNO  has  performed  research  into  using  intelligent  agents  to  generate 
behavior  that  can  be  used  both  for  training  [14]  and  decision  support  [15].  Intelligent  agents  can  greatly 
improve  the  functionality  of  virtual  training  systems  and  help  support  tactical  operations.  Commonly 
advanced  agent  systems  run  as  a  distributed  system  where  agent  behavior  is  performed  on  a  specific  agent 
simulation  platform  and  the  virtual  environment  is  simulated  on  a  separate  platform.  Again  the  question  on 
how  and  what  should  be  communicated  between  these  systems  is  not  trivial  to  answer. 

One  new  area  of  research  is  the  mediation  of  machine  expertise  with  human  users  through  natural 
communication  (e.g.  speech  and  dialog)  [16].  Such  advanced  man-machine  interfaces  (MMI)  can  greatly 
improve  the  usability  of  highly  automated  systems.  For  example,  TNO  is  working  on  Ashley.  Ashley  is  a 
novel  type  of  interface  that  offers  an  intuitive  way  to  communicate  with  many  machines  through 
intelligent  dialog  and  speech.  In  this  approach  it  is  essential  that  Ashley  can  reason  about  the  machines 
and  capabilities  that  she  represents  towards  the  user.  One  key  aspect  is  to  determine  how  machines  and 
Ashley  communicate  to  establish  what  Ashley  can  do  for  a  user.  Similar  issues  arise  with  other  advanced 
interfaces,  such  as  an  adaptive  task  allocation  interface  between  a  user  and  automated  support  systems. 

Internationally  there  is  much  related  work  being  done  on  man-machine  networked  operations.  In  [5]  an 
argument  is  put  forward  to  use  man-machine  network  simulations  to  investigate  the  benefits  of 
autonomous  systems  in  both  civilian  and  military  organizations.  Many  similar  studies  support  this 
argument  and  report  increased  need  for  autonomous  systems  to  improve  operations  and  reduce  casualties, 
see  for  example  [6], [7], [8]  and  [9].  However,  the  authors  of  [5]  do  not  go  into  detail  on  how  the  creation 
of  man-machine  simulations  can  be  supported.  Ziilch  and  Becker  [3]  present  an  integrated  system  that  can 
help  determine  the  optimal  leverage  of  men  and  machines  in  manufacturing  processes.  The  system  is 
supported  by  a  constructive  simulation  of  both  personnel  and  machines.  Such  a  constructive  approach  has 
the  advantage  that  all  simulation  is  done  in  one  monolithic  system  which  simplifies  the  development  of  the 
simulation.  The  drawback  is  that  it  is  computationally  limiting  to  model  such  a  complex  simulation  in  one 
monolithic  system.  Due  to  the  computational  limitation  the  resulting  simulation  is  therefore  inherently 
abstract  and  limited  in  detail.  In  [4]  an  example  of  a  human-machine  network  simulation  (air  MIDAS)  is 
described  to  investigate  errors  in  aviation  operations.  The  air  MIDAS  simulation  again  is  a  tightly  coupled 
system  of  environment  simulation,  behavior  models  and  system  models.  It  would  be  beneficial  if  parts  of 
the  system  could  be  re-used  (e.g.  the  behavior  models)  or  if  other  parts  could  be  flexibly  inserted. 

Summarizing,  man-machine  network  simulation  requires  expertise  from  multiple  disciplines  and  flexible 
integration  of  models,  which  remains  a  key  challenge.  With  the  M4E  approach  we  try  to  address  this  issue 
by  a)  facilitating  re-use  of  simulation  models,  b)  proposing  a  flexible  way  to  create  man-machine  network 
simulations  and  c)  preventing  researchers  to  have  to  start  from  scratch  by  offering  systems  that  can 
function  as  a  starting  point  to  develop  a  new  man-machine  network. 
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2.0  SIMULATING  MAN-MACHINE  NETWORKS 

To  analyze  complex  organization  such  as  military  units,  business  organizations,  and  crisis  organizations 
one  needs  to  at  least  take  into  account:  the  environment  of  the  organization,  the  information  that  is 
available  (for  instance  via  sensors),  the  information  systems  that  are  used,  and  of  course  the  human 
resources  of  the  organization.  This  essentially  describes  a  complex  man-machine  network  operating  within 
an  environment. 

To  create  a  simulation  for  this  analysis,  one  needs  the  following  components:  an  environment  simulation, 
a  sensor  simulation,  models  of  the  information  systems,  and  either  human  behavior  models  or  user 
interfaces  (for  human-in-the-loop  simulations).  All  these  components  are  placed  in  a  network  and  each 
network  node  is  connected  to  all  others,  see  Figure  1 . 


Figure  1 :  Man-Machine  network  components. 

There  are  many  kinds  of  simulated  environments  (e.g.  live,  virtual,  constructive,  games).  An  environment 
simulation  must  at  least  be  able  to  perform  simulated  commands  and  deliver  relevant  world  information  to 
other  network  nodes.  The  challenge  is  to  avoid  the  effort  of  creating  new  network  couplings  between  each 
simulated  environment  and  the  other  parts  of  the  human-machine  simulation  network. 

There  are  also  many  types  of  sensors  and  in  some  cases  one  ‘sensor’  consists  of  an  entire  network  of 
smaller  sensors.  The  key  requirement  of  a  sensor  in  a  man-machine  network  is  that  it  must  be  able  to  send 
sense  data  across  the  network.  However,  the  nature  of  sense  data  varies  greatly  among  sensors  (in  type  of 
data,  frequency  and  complexity).  The  challenge  is  therefore  to  flexibly  federate  sensors  with  the  other 
nodes  of  the  network. 

Information  system  simulations  again  are  available  in  great  variety.  At  TNO  we  have  a  strong  interest  in 
support  systems  for  operators  as  well  as  in  human  behavior  simulations.  Both  information  systems  are 
usually  modeled  in  the  form  of  intelligent  agents.  For  a  human-machine  network  simulation  an  intelligent 
agent  should  be  able  to  receive  information  and  produce  appropriate  actions.  In  other  words,  an  agent  must 
be  able  to  execute  relevant  tasks  and,  in  so  far  the  agent  represents  a  man  or  machine  that  is  part  of  a  team, 
it  must  be  able  to  communicate  about  those  tasks  with  other  agents  and  with  human  operators.  The 
challenge  therefore  is  to  create  a  way  of  communication  that  is  transparent  for  both  machine-machine 
connections  and  man-machine  connections. 

The  man-machine  interface  (MMI)  is  the  primary  way  for  human  users  to  access  the  network.  All  other 
components  are  part  of  the  same  “world”  from  the  perspective  of  the  user.  The  challenge  is  to  create  an 
MMI  that  can  adapt  its  domain  related  information  presentation  to  suit  the  expectations  of  professional 
users  while  keeping  consistent  those  parts  of  the  MMI  that  concern  domain  independent  interaction.  An 
example  of  a  domain  independent  interaction  could  be  changing  the  amount  of  support  the  user  wants  to 
receive  from  information  systems. 
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In  short,  we  are  trying  to  define  a  bridge  between  the  fields  of  environment  simulation,  machine 
perception,  intelligent  agents  and  man-machine  interfacing.  In  each  of  these  fields  there  are  (emerging) 
interoperability  standards.  However,  there  is  little  work  being  done  that  integrates  these  approaches.  We 
propose  the  M4E  approach  using  the  M4E  toolbox  to  construct  man-machine  networks. 

2.1  M4E  Approach 

To  develop  the  M4E  approach  we  take  a  representative  research  challenge,  and  try  to  define,  partly 
drawing  on  lessons  learned,  what  support  can  be  created  for  the  researcher  that  is  developing  a  man- 
machine  network  simulation.  In  the  first  iteration  we  focus  on  compound  protection  as  a  representative 
field  of  research.  For  this  case  the  challenge  is  to  develop  a  man-machine  network  that  can  be  used  to 
explore  different  levels  of  automation  and  support  for  an  operator  tasked  with  compound  protection. 

General  Scenario  Description 

There  is  a  crowd  near  the  fence  of  a  compound.  A  small  number  of  people  behave  suspiciously  in  the 
sense  that  they  enter  the  forbidden  area.  These  people  are  detected  by  sensors.  A  threat  detector  support 
system  (agent)  signals  the  suspicious  behavior  and  informs  a  command  support  system  (agent)  and  the 
human  operator  (via  a  MMI).  The  human  operator,  supported  by  the  command  agent,  takes  action  and, 
also  via  the  MMI,  commands  a  small  quick  reaction  force  (QRF)  to  arrest  the  suspicious  people.  The  QRF 
is  simulated  and  its  behavior  is  controlled  by  a  behavior  simulation  (agent). 

Environment 

The  environment  consists  of  a  compound,  which  is  surrounded  by  a  fence  with  two  gates.  Around  the 
fence  there  is  an  area  where  no  (external)  people  are  allowed.  See  below. 


Figure  2:  Compound  overview. 


Units 

•  A  crowd  outside  the  gate,  from  which  two  persons  (armed  with  explosives)  try  to  intrude  the 
compound. 

•  QRF,  which  consists  of  a  team  of  4  that  can  arrest  the  intruders. 

•  Sensor(s),  located  at  the  compound.  It  can  distinguish  positions  of  contacts  in  the  area  outside  the 
compound  till  10  meters  outside  the  forbidden  area. 

•  Threat  Detection  support  system,  with  whom  the  human  operator  through  the  MMI  has  made 
arrangements  on  when  to  inform  the  MMI.  This  system  passes  a  contact’s  position  to  the  MMI 
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and  can  aid  in  its  identification  (based  on  its  intrusion  level  and  whether  it  is  an  own  force 
yes/no).  We  distinguish  four  identifications  (hostile,  friendly,  neutral,  unknown). 

•  Command  support  system,  that  can  support  the  operator  in  its  task  to  decide  on  how  the  handle 
intruders. 

•  Human  operator,  who  is  responsible  for  the  protection  if  the  compound.  The  operator  identifies 
contacts  and  decides  how/which  intruders  to  arrest. 

Exploration  of  Different  Levels  of  Support 

In  this  case  we  want  to  compare  two  levels  of  support  for  the  operator.  We  offer  a  low  level  of  support  in 
which  the  workload  of  protecting  the  compound  rests  mainly  with  the  operator.  Alternatively  in  the 
condition  of  high  level  support  the  operator  is  supported  in  threat  detection  and  resource  assignment. 

In  the  Tow  level  of  support’  condition  the  Threat  Detection  support  will  deliver  a  limited  set  of 
alternatives  to  the  operator.  Given  the  task  ‘threat  detection’  or  perhaps  better  ‘identification’,  the  system 
will  provide  the  positions  of  contacts,  together  with  the  possible  identifications  (unknown,  neutral, 
friendly,  hostile).  The  operator  must  then  decide  which  alternative  it  will  be. 

In  the  ‘high  level  of  support’  the  Threat  Detection  support  will  execute  the  best  alternative.  This  means  it 
will  assign  an  identification  to  each  contact  which  can  be  shown  on  the  MMI. 

In  the  Tow  level  of  support’  the  Command  support  system  will  deliver  a  limited  set  of  alternatives  to  the 
operator.  Given  the  task  ‘command’  or  perhaps  better  ‘contact  handling  by  arresting’,  the  system  will 
provide  the  MMI  for  detected  contacts  the  possible  QRF  assignments.  The  operator  must  then  decide 
which  assignment  to  make. 

In  the  ‘high  level  of  support’  the  Command  support  system  will  execute  the  best  alternative  if  the  operator 
approves.  This  means  it  will  provide  the  MMI  with  the  best  contact-QRF  assignment  and  the  option  to 
confirm  this  assignment. 

With  a  human-machine  network  at  our  disposal  that  can  accommodate  this  scenario  and  the  different 
levels  of  support  we  can  perform  a  systematic  experiment  into  which  level  of  support  is  suitable  under 
which  conditions  for  the  operator. 

Even  with  a  relative  simple  case  as  the  compound  scenario  case,  the  amount  and  types  of  data  flowing 
across  the  man-machine  network  becomes  complex  quickly,  see  figure  3.  This  underscores  the  helpfulness 
of  developer  tools  in  creating  such  networks.  In  the  following  section  we  will  describe  what  tools  TNO  is 
creating  to  support  the  development  of  such  a  network. 
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Figure  3:  M4E  Compound  protection  case  data  flow. 


3.0  TOOLS  IN  THE  TOOLBOX 

Within  the  M4E  project  we  aim  to  create  man-machine  network  developer  tools  that  facilitate  the  creation 
of  such  networks  for  research.  In  this  section  we  describe  the  toolbox  that  contains  the  results  of  our  effort. 

3.1  Simulation  Tools 

The  intent  of  TNO  is  to  provide  an  environment  for  easy  development  of  simulated  environments  enriched 
with  additional  components  such  as  Artificial  Intelligent  (AI)  Behavior.  The  AI  can  be  created  close  to  the 
virtual  environment  or  as  a  separate  part.  A  separate  implementation  has  the  advantage  of  being 
independent  of  computing  cycles  of  the  simulator  and  gives  the  freedom  of  choosing  a  programming 
language.  The  disadvantage  is  that  you  have  to  transport  all  information  (commands  and  reports)  between 
the  simulated  environment  and  the  AI  platform. 

The  interfaces  used  in  M4E  aim  to  be  standardized  in  a  way  to  allow  an  easy  swap  between  simulation 
environments  (e.g.  VR-Forces2,  VBS23)  and  AI  (e.g.  Jadex  [10]).  It  is  desired  to  make  this  interface  based 
on  existing  standards. 

HLA  is  chosen  for  the  M4E  interface.  The  HLA  RPR  FOM  is  a  data  exchange  standard  (SISO-STD-OOl- 
1999)  that  has  received  many  followers  in  the  field.  While  it  is  a  good  starting  point  to  define  the  data  that 
needs  to  be  shared  between  simulations,  it  was  never  intended  for  a  broader  scope  (including  AI  or  MMI) 
or  a  more  flexible  approach.  We  propose  the  take  this  FOM  as  a  starting  point  and  to  extend  it  with  man- 
man-machine-machine  network  specific  additions. 

“  http://www.mak.com/ 

3 

http://www.bisimulations.com/ 
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HLA  is  also  a  beneficial  choice  because  the  HLA  is  already  a  standard  in  TNO  simulation  federations. 
Moreover,  the  AI  architecture  under  consideration  is  being  prepared  with  an  HLA  interface  (in  the  form  of 
a  combination  between  Pogamut4  and  Jadex).  The  information  exchanged  in  M4-networks  will  be  secured 
in  the  M4E-FOM. 

HLA  Related  Choices 

HLA  is  available  in  three  flavors: 

•  HLA  1.3 

•  HLA  IEEE  1516 

•  HLA  Evolved 

HLA  1.3  is  the  first  implementation  of  HLA.  With  HLA  IEEE  1516  a  number  of  problems  found  during 
practical  use  of  HLA  1.3  were  solved  and  it  was  accepted  as  a  standard.  The  newest,  HLA  Evolved, 
extends  HLA  with  amongst  others  modular  FOM  and  dynamic  link  compatibility. 

For  optimal  flexibility  HLA  evolved  is  the  way  to  go  as  this  allows  federates  to  only  implement  and  use 
the  part  of  the  FOM  that  they  require.  But  as  this  is  a  very  new  standard  many  applications  do  not  comply 
to  HLA  evolved  yet  (e.g.  VR-Forces). 

For  practical  reasons  the  first  iteration  of  M4E  will  therefore  use  HLA  1.3.  For  example,  Pogamut  is  being 
developed  with  the  portico  RTI,  only  available  in  1.3.  VR-Forces  is  available  in  1.3  and  IEEE  1516.  VR- 
Forces  is  a  product  of  MaK,  so  we  build  on  the  Mak  RTI.  In  the  future  M4E  will  migrate  to  HLA  evolved 
depending  on  the  available  support. 

HLA  allows  for  objects  and  interactions.  Interactions  are  once-only  while  objects  stay  alive  until  deleted. 

For  tasks  and  reports  both  types  are  possible.  When  you  use  objects,  the  status  of  the  task  can  be  updated. 
For  example,  someone  sends  a  report  and  someone  else  updates  it  with  executing,  or  done.  But  this  will 
add  a  time-consuming  ownership  management  to  the  system.  It  can  also  be  done  with  interactions.  The 
interactions  then  need  an  ID  so  the  response  can  refer  to  this  ID.  For  M4E  the  choice  is  made  of  sending 
tasks  and  reports  over  interactions 

M4E  Specific  FOM  Definitions 

For  M4E  we  have  define  the  following  additional  classes  to  the  RPR  FOM: 

Table  1 :  Additional  FOM  classes. 


Object  Classes 

Interaction  Classes 

Task  Coordination 

T  askDescription, 
AgentConfiguration, 
M4ET  askSetting 

RequestT  askAllocation, 

RefuseT askAlloc  ation,  AgreeT askAlloc  ation 

Task  Specific 

Contact, 

QRFPersonnel, 

QRFAssigment 

Detection,  ThreatLevelOptions, 

ThreatLevelDecision, 
QRFAssignmentOptions, 
QRFAssigmentDecision,  Order,  MoveTo 

4 


http://artemis.ms.mff.cuni.cz/pogamut 
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Depending  on  the  specific  man-machine  network  and  the  shared  task  of  the  man-machine  team  there  will 
always  be  task  specific  adaptations  that  must  be  made.  This  is  an  unavoidable  fact  that  is  part  of  the  reality 
of  doing  novel  research.  However,  communication  about  task  coordination  and  collaboration  is  a  part  of 
the  data  exchange  that  we  can  facilitate  by  using  the  Task  Coordination  classes. 

3.2  Man  Machine  Interface  Tools 

Humans  have  been  interfacing  with  simulations  since  the  advent  of  simulation.  However,  few  human- 
machine  interfaces  have  been  developed  that  are  specifically  useful  in  shaping  man-man-machine-machine 
collaboration.  Often  simulation  interfaces  are  strictly  domain  orientedand  offer  little  support  for  a 
structured  or  comprehensive  exploration  of  different  collaboration  concepts. 

Experimenting  with  man-machine  teams  often  means  (re)considering  the  workload  division  between  man 
and  machine.  A  very  effective  research  tool  is  the  ability  to  change  such  workload  divisions  across 
experimental  conditions.  However,  simple  changing  the  workload  division  means  that  the  user  will  have  a 
different  interaction  with  the  machine.  When  the  workload  division  changes,  the  interface  should  adapt 
itself  to  the  new  type  of  interaction  that  results.  So  there  are  two  main  requirements  1)  the  workload 
division  should  be  configurable  (most  likely  by  the  researcher),  and  2)  the  user  interface  must  adapt  to  a 
new  workload  division. 

Within  M4E  we  recognize  that  for  most  research  projects  there  will  always  be  a  domain  specific  user 
interface  required.  However,  we  can  develop  a  reusable  user  interface  that  allows  changes  in  the  division 
of  work.  Additionally,  because  we  also  develop  tools  for  communication  (with  the  M4E  addition  to  the 
RPR  FOM)  and  templates  for  intelligent  systems  we  can  ensure  that  such  work  division  changes  are  easily 
supported  in  communication  architecture  and  intelligent  system  design. 

For  the  collaboration  interface  we  envision  a  three  stage  user  interface.  These  stages  range  from  abstract 
and  simple  to  extensive  and  intuitive.  Depending  on  the  research  project  one  only  needs  a  work  division 
change  when  the  researcher  demands  it  (between  experimental  conditions).  In  other  cases  the  researcher 
may  want  to  experiment  with  real-time  dynamic  changes  in  work  division  by  the  operator.  By  have  three 
stage  of  the  collaboration  interface  one  can  pick  and  choose. 

The  first  stage  collaboration  interface  is  a  simple  pie  chart  which  shows  the  amount  of  support  that  the 
operator  receives  from  the  system,  see  table  2. 

The  second  stage  collaboration  interface  shows  the  tasks  that  need  to  be  performed  by  the  operator  and  the 
available  support  for  those  tasks.  The  researcher  or  operator  can  specify  how  much  support  is  required  for 
each  task. 

The  third  stage  collaboration  interface  is  a  fully  fledged  virtual  assistant  that  can  notify  and  explain  the 
work  division  options.  The  virtual  assistant  is  voice  controlled  and  the  operator  can  speak  his  work 
division  desires  and  the  assistant  will  execute  what  is  ordered. 
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Table  2:  Three  stage  user  interface  for  man-machine  collaboration. 


Level  1 :  high-level 
assistance  &  control. 


Level  2:  low-level 
assistance  &  control 


Level  3.  Collapsed 
assistant. 

Placement 
represents  amount 
of  support  (  high  = 
high;  low  =  low). 

Assistance  available 
by  single  user  action 
(click/tap/speech). 
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3.3  Sensor  Simulation  Tools 

Sensors  have  often  been  developed  in  isolation.  Much  work  still  remains  to  be  done  on  developing  a 
(military)  standard  to  connect  sensors  in  a  network.  We  propose  to  integrate  sensor  and  simulation 
approaches  as  much  as  possible.  We  develop  sensor  models  that  produce  HLA-RPR-FOM  compliant  output. 

TNO  has  developed  the  TActical  sensor  network  TEst  bed  (TASTE)  [2].  TASTE  is  a  simulation  platform 
that  can  be  utilized  to  simulate  novel  sensors.  TASTE  started  out  as  a  software  tool  for  specifying  and 
deploying  unattended  ground  sensors  (UGS)  in  a  composition  which  the  commander  assumes  will  suit  his 
needs  the  best.  Within  TASTE  different  sensor  types  such  as  acoustic,  magnetic,  seismic,  radar  and  IR 
imaging  sensors  can  be  deployed  virtually,  and  their  individual  and  combined  performances  analyzed.  It  is 
therefore  well  suited  as  a  platform  to  investigate  sensor  networks.  Sensors  can  be  deployed  in  scenarios  for 
neutral,  friendly  and  enemy  movements  and  a  set  of  typical  environmental  conditions  can  be  selected  as 
input  to  the  simulator.  The  taste  platform  is  offered  in  M4E  as  a  convenient  way  to  connect  sensor 
simulations  to  other  simulations,  intelligent  agents  and  user  interfaces. 

3.4  Intelligent  Agents  Tools 

Intelligent  agents  are  a  promising  area  that  can  contribute  much  to  man-machine  network  simulations.  Up 
until  now  most  of  the  agent  computing  platforms  have  been  developed  in  isolation  (e.g.  ACT-R5,  SOAR6, 
JADEX).  We  are  extending  an  existing  agent  framework  to  integrate  more  easily  with  simulations  and 
sensors.  This  way  simulation  developers  no  longer  have  to  start  from  scratch  when  experimenting  with 
intelligent  systems  in  a  man-machine  network. 

We  have  started  out  with  JADEX  as  the  agent  platform  based  on  past  experience  and  the  fact  that  it  is  an 
open  platform.  A  recurring  problem  with  agent  development  is  that  the  link  between  an  agent  and  the 
simulated  world  is  often  a  connection  along  which  information  flows  that  is  not  standardized  in  any  way. 
We  have  extended  JADEX  with  a  HLA  connection  and  a  way  to  translate  HLA  messages  to  JADEX 
messages.  Agents  within  JADEX  now  have  a  means  of  communicating  across  HLA.  Additionally  we  have 
developed  an  agent  template  that  can  process  the  M4E  FOM  messages.  This  enables  an  agent  to 
automatically  respond  to  changes  in  work  division.  If  more  intelligent  support  is  needed,  (sub)tasks  are 
transferred  to  the  agent  and  the  agent  changes  the  way  in  which  it  communicates.  This  way  support 
system,  user  interface  and  simulation  all  can  work  in  concert  to  adapt  the  simulation  to  different  ways  of 
working. 

We  have  created  a  flexible  way  of  dealing  with  (dynamic)  work  division  between  man  and  machine. 
Central  to  this  approach  is  the  shared  task  (or  shared  goal)  of  the  man-machine  network.  Both  the  machine 
side  as  the  human  side  must  agree  on  a)  what  it  is  that  needs  to  be  done,  and  b)  how  the  task  can  be 
decomposed  and  delegated.  For  this  we  propose  a  three  type  categorization  of  tasks. 

Table  3:  Different  task  classes. 


Task  Type 

Description 

Atomic 

Atomic  tasks  can  not  be  subdivided  and  are  always  performed  by  either  a  man  or  machine.  When 
delegated  they  are  delegated  in  their  entirety. 

Mixed 

Mixed  tasks  is  collection  of  subtasks  that  can  be  a  mix  of  atomic  and  M4E  subtasks.  Usually  this  is 
the  type  associated  with  the  shared  goal  or  overall  task  of  the  man  machine  team. 

M4E 

This  type  of  task  is  structured  in  three  distinct  subtasks:  suggest,  decide  and  act.  This  type  of  task 
can  be  executed  in  a  shared  fashion  between  man  and  machine.  The  subtasks  must  be  performed  in 
order. 

5  act-r.psy.cmu.edu 

6  sitemaker.umich.edu/soar 
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The  M4E  task  type  allows  us  to  create  a  system  of  collaboration  while  still  leaving  room  for  other  types  of 
task  structures.  The  research  retains  freedom  of  choice  in  this  regard.  However,  M4E  offers  advanced 
collaboration  support  for  tasks  that  are  structured  along  the  M4E  type  structure. 

Usually,  when  man-machine  collaboration  is  discussed  the  10  point  scale  of  Parasuraman,  et  al.  [17]  is 
referenced  to  denote  the  amount  of  work  assigned  to  man  or  machine.  We  posit  that  this  scale  confuses 
work  distribution  and  communication  about  the  work  itself.  For  the  M4E  task  type  we  have  created  a  new 
work  division  concept  that  explicitly  decouples  work  division  from  communication  about  the  work. 


Figure  3:  M4E  task  type. 


Figure  3  shows  that  a  M4E  task  needs  input  (M)  to  activate  and  results  in  output  (P).  At  this  level  the 
choice  is  only  whether  or  not  to  share  the  output  with  the  other  members  of  the  man-machine  network. 
However,  if  we  observer  one  layer  deeper  we  can  see  that  the  output  from  the  suggest  sub -task  (N)  and  the 
decide  sub-task  (O)  can  potentially  also  be  communicated.  For  example,  consider  an  M4E  task  that  is 
shared  between  man  and  machine  where  the  machine  is  responsible  for  the  suggest  sub -task  and  the 
human  for  the  decide  and  act  sub-tasks.  In  this  case  the  machine  necessarily  needs  to  communicate  N  to 
the  human  so  that  the  decide  and  act  sub-tasks  may  be  executed.  Output  O  is  not  communicated  at  all 
since  this  output  only  exists  in  the  mind  of  the  human.  The  human  finally  produces  output  P.  This  example 
shows  that  communication  about  the  task  follows  naturally  from  the  division  of  work.  This  makes  it 
possible  to  prepare  an  intelligent  agent  “template”  that  follows  this  structure  and  automatically  adapts 
communication  rules  to  changes  in  work  division  in  the  human-machine  network. 

3.5  Domain  Knowledge  Tools 

One  of  the  key  issues  when  building  a  man-machine  network  is  to  make  sure  that  all  personnel  and 
machines  collaborate  on  the  same  shared  mission.  However,  in  a  man-machine  network  simulation  a  lot  of 
different  parts  need  to  work  in  harmony.  This  poses  a  risk  that  the  different  parts  may  not  agree  on  the 
shared  mission  and  start  to  work  past  each  other.  In  the  M4E  toolbox  we  provide  a  generic  way  to 
structure  the  mission  or  task  under  investigation  that  enables  simulated  environment,  sensor,  agents  and 
man-machine  interfaces  to  work  in  concert  and  keep  them  aligned  on  the  same  mission. 

Because  HEA  supports  persistent  shared  data  models  it  is  relatively  straightforward  to  publish  a  task- 
ontology  throughout  the  man-machine  network.  We  have  defined  the  following  object  classes  for  this 
purpose. 
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Table  4:  This  table  shows  the  classes  used  to  share  task  and  work  division  information. 


Classname 

Attribute 

TaskDescription 

Taskid 

Description 

Type 

SubTasks 

AgentConfiguration 

Agentid 

Name 

SupportedTask 

M4ET  askSetting 

Taskid 

M4EType 

Agent  lid 

Agent2Id 

The  task  description  objects  define  the  shared  mission  (main  task)  and  all  the  underlying  subtasks.  The 
agent  configuration  objects  are  used  to  signal  which  man-machine  team  members  can  perform  which 
tasks.  Finally,  the  task  setting  objects  are  used  to  signal  across  the  entire  network  how  the  work  is  divided 
at  any  moment.  The  machine  part  (i.e.  the  agents)  of  the  man-machine  network  can  assimilate  this  task 
ontology  without  translation.  The  human  part  has  to  be  supported  by  the  MMI  to  provide  insight  in  the 
task.  Additionally,  the  MMI  will  need  to  translate  human  capabilities  in  terms  of  agent  configuration  in 
order  for  the  machine  part  to  know  what  the  human  part  can  do. 


4.0  DISCUSSION  AND  FURTHER  WORK 

In  this  paper  we  have  described  the  tools  that  are  the  part  of  the  M4E  toolbox.  As  we  stated  in  the 
introduction,  man-machine  research  project  often  start  from  scratch  or  with  very  little  re-use  of  systems. 
As  a  direct  result  from  M4E  there  are  now  HLA  object  classes  defined  that  offer  a  way  to  communicate 
about  task  coordination  and  collaboration  suitable  for  any  domain.  These  have  been  implemented  and  are 
ready  to  be  (re)used  across  multiple  research  projects. 

If  there  is  little  re-use  then  lessons  learned  are  not  anchored  in  technology  across  projects.  M4E  allows 
better  transfer  of  lessons  learned  about  work  division  by  defining  an  information  exchange  contract  with 
respect  to  man-machine  collaboration.  If  M4E  is  successful  and  multiple  project  use  the  information 
exchange  contract  and  extend  it  then  there  is  even  more  potential  to  “stand  on  the  shoulders  of  others”. 

We  put  forward  in  the  introduction  that  a  lack  of  standards  to  federate  heterogeneous  models  is  impairing 
innovative  new  simulation  concepts.  We  have  shown  that  in  the  M4E  vision  we  aim  to  leverage  the 
strength  of  specific  models  by  composing  a  system  from  distributed  models.  Furthermore,  we  propose  to 
improve  that  strength  by  specifying  standards  that  regulate  how  models  can  work  together  (such  as  in  the 
case  of  work  division). 

Developing  a  man  machine  network  is  inherently  complex  and  therefore  benefits  from  developer  tools  that 
speed  up  their  creation.  With  M4E  we  have  provided  tools  for  simulation,  sensor  simulation,  man-machine 
interfacing,  intelligent  systems  design  and  domain  knowledge  sharing  to  facilitate  the  creation  of  such  a 
network.  This  will  reduce  the  burden  on  the  researchers  and  developers  in  creating  the  necessary 
experimental  setup  that  they  require  to  create  insight  in  man-machine  networked  operations.  The  toolbox 
will  shorten  development  time  and  will  help  keep  the  focus  on  the  operational  research  questions. 
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